Building Your Own OpenTelemetry Stack: A .NET Developer's Tutorial with Jaeger, Prometheus, and the OpenTelemetry Collector

In a previous post I wrote about how to get started with deploying the OpenTelemetry collector to AWS Fargate. But doing this every time to test you metrics and traces is not very efficient. Besides that it is also hard to debug. In a world of devops we want a fast feedback loop. So how can we do this?

In this blog post I will show you how to setup a local OpenTelemetryCollector stack to test your metrics and traces before deploying them.

We create this from scratch. So we will start with a simple application that will generate some metrics and traces. After that we will add the OpenTelemetry collector to collect the metrics and traces. And finally we will use Jaeger for the tracing and add Prometheus to collect the metrics.

The application #

In this example we will use a simple ASP.NET Core application. The so called Weather api.

To create a new project execute the following command:

mkdir src && cd src
dotnet new webapi -o WeatherApi

Next we will add the OpenTelemetry packages to the project.

cd WeatherApi
dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

First we need to create a meter to collect the metrics. Create a new file WeatherMeter.cs and add the following code to the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


using System.Diagnostics.Metrics;

namespace WeatherApi;

public class WeatherMeter
{
    private readonly Counter<int> _weatherForecastRequestCounter;

    public WeatherMeter()
    {
        var meter = new Meter(nameof(WeatherMeter));
        Name = meter.Name;
        _weatherForecastRequestCounter = meter.CreateCounter<int>("get_weather_forecast_request_counter");
    }

    public static string Name { get; private set; } = null!;

    public void GetWeatherForecast() => _weatherForecastRequestCounter.Add(1);
}

Here we create a new meter and a counter. The counter will be used to count the number of requests to the WeatherForecastController. The GetWeatherForecast method is a wrapper around the counter and will ensure that the counter will be incrementend by one. The Name property will be used to register the meter with the OpenTelemetry collector.

Next we initialize an activity source to create traces. Create a new file Diagnostics.cs and add the following code to the file.

1
2
3
4
5
6
7
8


using System.Diagnostics;

namespace WeatherApi;

public static class Diagnostics
{
    public static readonly ActivitySource WeatherServiceActivitySource = new("weather-service");
}

Now you need to edit the Program.cs file to add the OpenTelemetry packages to the application. We will also register the WeatherMeter and the ActivitySource so metrics and traces will be collected. Add the following code to the CreateHostBuilder method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


...

// Register WeatherMeter
builder.Services.AddSingleton(new WeatherMeter());

// Add OpenTelemetry
var otelEndpoint = Environment.GetEnvironmentVariable("OTEL_COLLECTOR_ENDPOINT");
if (string.IsNullOrEmpty(otelEndpoint))
{
    otelEndpoint = "http://localhost:4317";
}

var resourceBuilder = ResourceBuilder.CreateDefault().AddService("weather-service")
   .AddTelemetrySdk();

builder.Services.AddOpenTelemetry()
   .ConfigureResource(resource => resource.AddService("weather-service"))
   .WithTracing(tracing => tracing
        .AddConsoleExporter()
        .AddAspNetCoreInstrumentation()
        .AddSource("weather-service")
        .AddOtlpExporter(exporter =>exporter.Endpoint = new Uri(otelEndpoint)))
   .WithMetrics(metrics =>
    {
        if (builder.Environment.IsDevelopment())
        {
            metrics.AddConsoleExporter();
        }
        metrics.AddMeter(WeatherMeter.Name)
           .AddOtlpExporter(exporter => exporter.Endpoint = new Uri(otelEndpoint));
    });

var app = builder.Build();
....

This will add the OpenTelemetry packages to the application. It will also configure the OpenTelemetry collector to send the metrics and traces to the local collector.

After that we need to add some code to the WeatherForecastController to generate some metrics and traces.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


using Microsoft.AspNetCore.Mvc;

namespace WeatherApi.Controllers;

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
    private static readonly string[] Summaries = new[]
    {
        "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
    };

    private readonly ILogger<WeatherForecastController> _logger;
    private readonly WeatherMeter _weatherMeter;

    public WeatherForecastController(ILogger<WeatherForecastController> logger, WeatherMeter weatherMeter)
    {
        _logger = logger;
        _weatherMeter = weatherMeter;
    }

    [HttpGet(Name = "GetWeatherForecast")]
    public IEnumerable<WeatherForecast> Get()
    {
        using var activity = Diagnostics.WeatherServiceActivitySource.StartActivity("GetWeatherForecast");
        _weatherMeter.GetWeatherForecast();
            
        _logger.LogInformation("GetWeatherForecast called");
        return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
                TemperatureC = Random.Shared.Next(-20, 55),
                Summary = Summaries[Random.Shared.Next(Summaries.Length)]
            })
           .ToArray();
    }
}

On line 26 we start a new activity. This will create a new trace. On line 27 we increment the counter using the method we created earlier in the WeatherMeter.

The application is now ready to create metrics and traces and is able to send them to the open telemetry collector.

Next thing to do is to dockerize the application. Create a Dockerfile to the project and add the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /src
COPY ["WeatherApi.csproj", "./"]
RUN dotnet restore "WeatherApi.csproj"
COPY . .
WORKDIR "/src/"
RUN dotnet build "WeatherApi.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "WeatherApi.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WeatherApi.dll"]

Now the application is ready to run in a container. To test this we can run the following command:

docker build -t weatherapi .
docker run -p 8080:80 weatherapi

If you head over to http://localhost:8080/weatherforecast you should see the weather forecast data.

Docker compose #

Now we have a working application, we can start with the OpenTelemetry collector. To make this easy we will use docker compose. This will allow us to run multiple containers at the same time.

Create a new file called docker-compose.yml and add the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


version: '3.8'
services:
  weatherapi:
    image: weatherapi
    build:
      context: src/
      dockerfile: Dockerfile
    environment:
        - OTEL_COLLECTOR_ENDPOINT=http://otel-collector:4317
    ports:
        - "8080:80"
  
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./config/otel-collector-config.yaml:/etc/otel-collector-config.yaml
    command: ["--config=/etc/otel-collector-config.yaml"]
    ports:
      - "4317:4317"
      - "4318:4318"
      - "13133:13133"

  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "14250:14250" # Port for gRPC
      - "14268:14268"
      - "16686:16686" # Port for User Interface
  
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./config/prometheus-config.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090" # Port for User Interface

In this docker compose file we define 4 services. The weatherapi, the otel-collector, Jaeger and Prometheus. The weatherapi is the application we created earlier. The otel-collector is for retrieving traces and metrics. The jaeger service is the Jaeger UI. The prometheus service is the Prometheus UI. There are some references to configuration files. We will create these files in the next steps. On line we set the OTEL_COLLECTOR_ENDPOINT environment variable to override the default endpoint. This is needed because the default endpoint is not accessible from the weatherapi container.

Configuring the OpenTelemetry collector and Prometheus #

With all services defined the last thing to do is to configure the OpenTelemetry collector. To do this we need to create a new file called otel-collector-config.yaml. Add the following content to the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    verbosity: detailed
  otlp/jaeger: # Jaeger otlp collector
    endpoint: jaeger:4317 
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"
    path: "/health/status"
    check_collector_pipeline:
      enabled: true
      interval: "5m"
      exporter_failure_threshold: 5


processors:
  batch:

service:
  telemetry:
    logs:
      level: "debug"
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/jaeger]
      processors: [batch]
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]
      processors: [batch]
    logs:
      receivers: [otlp]
      exporters: [logging]
      processors: [batch]

The most interesting of this part is the jaeger exporter. This one is prefi

Prometheus #

To get metrics in to prometheus we need to configure the prometheus service. To do this we need to create a new file called prometheus-config.yml. Add the following content to the file:

1
2
3
4
5


scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 5s
    static_configs:
      - targets: ['otel-collector:8889']

Running the application #

To run everything we can run the following command:

docker-compose up -d

To see if everything is running we can run the following command:

docker-compose ps

This should see something similar like this:


CONTAINER ID   IMAGE                                    COMMAND                  CREATED        STATUS                            PORTS                                                                               NAMES
8550dc8df46b   otel/opentelemetry-collector-contrib     "/otelcol-contrib --…"   13 hours ago   Up 13 hours                       0.0.0.0:4317->4317/tcp, 0.0.0.0:55680->55680/tcp, 55678-55679/tcp                   weatherapi-otel-collector-1
cd22abb5b8df   jaegertracing/all-in-one                 "/go/bin/all-in-one-…"   13 hours ago   Up 13 hours                       5775/udp, 5778/tcp, 14250/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp   weatherapi-jaeger-1
1e8e94be8a21   weatherapi                               "dotnet WeatherApi.d…"   13 hours ago   Up 13 hours                       80/tcp, 443/tcp                                                                     weatherapi-weatherapi-1
e1050c408cb3   prom/prometheus                          "/bin/prometheus --c…"   13 hours ago   Up 13 hours                       0.0.0.0:9090->9090/tcp                                                              weatherapi-prometheus-1

View traces and metrics #

If you now head over to http://localhost:8080/weatherforecast you should see a generated weather forecast.

Open a new tab and go to http://localhost:16686, you should see the Jaeger UI. In the service dropdown menu, select weather-service, after that you should see the traces appear on each request made to the weather forecast service.

If you head over to http://localhost:9090 you should see the Prometheus UI. If you search for weather_requests_total you should see the counter we created earlier. If you click on the graph tab you should see a graph of the counter.

You can find the complete code on my github repo